Virtual Window Device And Method For Operating A Virtual Window Device

ABSTRACT

A virtual window device, particularly for use in an airplane, includes a bezel, or a frame, with a display arranged within the bezel, or within the frame; a distance sensor unit configured to produce at least one first distance signal indicating a first distance to an object; a first camera unit configured to capture a live video of a real scene; and a computing unit configured to select, based at least on the at least one first distance signal, a section of the live video of the real scene captured by the first camera unit; wherein the computing unit is further configured to control the display to display the selected section.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 15/689,458, filed on Aug. 29, 2017, the disclosure of which ishereby incorporated herein by reference.

FIELD OF THE INVENTION

The present invention pertains to a virtual window device and a methodfor operating a virtual window device. A virtual window device can beconsidered to be any system or apparatus that mimics an actual window,e.g. an unblocked or transparently sealed opening within a wall, withoutthere being an actual opening in said wall. The picture or scene thatthe virtual window device displays or projects to achieve this may becalled a virtual window.

Although applicable for any kind of wall, the present invention and thecorresponding underlying problems will be explained in further detail inconjunction with an external wall of an aircraft. Similarly, however,the present invention may be applied to, or used in, rooms, especiallywindowless rooms, in particular windowless rooms in vehicles, forexample windowless rooms in boats, airplanes, submarines and so on. Thepresent invention may also be applied to, or used in, rooms ofbuildings, in particular such rooms that have no actual windows, forexample, subterranean rooms.

BACKGROUND OF THE INVENTION

Windows do not only provide a person on one side of the window withawareness of the events on the other side of the window, they may alsoimprove a level of comfort that a person experiences within the room orspace provided with the window. In many cases, however, because ofstructural or other considerations, an actual window cannot be providedat all, or without large effort or causing unwanted additional effects.In such cases virtual windows can be provided that mimic actual windows.In this way, for example, a sturdier wall, without an opening or awindow bezel, may be provided while still affording many advantages ofactual windows.

U.S. Pat. No. 7,880,739 B2 describes a virtual window with simulatedparallax and field of view change.

BRIEF SUMMARY OF THE INVENTION

It is an idea of the present invention to provide an improved virtualwindow device that mimics actual windows especially closely.Particularly, it is an idea of the invention to improve the seamlessnessand the impression of reality of the scene displayed by the virtualwindow device. Another idea is to provide a virtual window device withbetter handling of a situation in which multiple persons are adjacent tothe virtual window device.

According to a first aspect of the invention, a virtual window devicecomprises a bezel or a frame with a display arranged within the bezel orthe frame; a distance sensor unit configured to produce at least onefirst distance signal indicating a first distance to an object; a firstcamera unit configured to capture a live video of a real scene; and acomputing unit configured to select, based at least on the at least onefirst distance signal, a section of the live video of the real scenecaptured by the first camera unit; wherein the computing unit is furtherconfigured to control the display to display the selected section of thelive video of the real scene captured by the first camera unit.

The object to which the first distance is determined and indicated canbe an object with a predetermined pattern, in particular a person or apart of a person, especially a face of a person.

The first distance indicated by the first distance signal can be adistance between the object and a predetermined point closely related tothe distance sensor unit, for example, an antenna of the distance sensorunit. The first distance indicated by the first distance signal maytherefore be a distance between the object and the distance sensor unitor a sub-unit of the distance sensor unit.

According to a second aspect of the present invention, a vehicle with avirtual window device according to the present invention is provided.The vehicle may in particular be an airplane, a train, or a boat.

According to a third aspect of the present invention, a method foroperating a virtual window device is provided, the method comprising thesteps of: determining a distance between an object and a referencepoint; capturing a live video of a real scene; selecting, based at leaston the determined distance, a section of the captured live video of thereal scene; and controlling a display of a virtual window device todisplay the selected section.

The object to which the distance is determined and indicated can be anobject with a predetermined pattern, in particular a person or a part ofa person, especially a face of a person.

The reference point may, in particular, be a point arranged at, oradjacent to, the display of the virtual window device, for example at ageometrical center of the display, e.g. its center of mass. Thereference point may alternatively be located at the location of adistance sensor element.

One underlying idea of the present invention is to provide to a personon one side of the virtual window device with the impression that theperson does not perceive a virtual window but an actual window to ascene located on the other side of the virtual window device. Inparticular, it is aimed to provide a virtual window device such, and/orto operate a virtual window device such, that the scene perceived by theuser may change with the distance of the user to the virtual windowdevice and/or with the orientation of the face of the user compared tothe virtual window device in the same way as the actual view through anactual window would change for the user, if the display of the virtualwindow device was an actual window.

For example, a room, or a vessel may be designed without actual windows,i.e. to be able to provide a continuous armored wall of the room orvessel. Using the present invention, the people inside the room orvessel are enabled to freely naturally perceive the environments of theroom or vessel. In some embodiments, augmented reality objects may beadded to the virtual window displayed.

One great advantage is that, a live video of the real scene is captured,of which only a section is displayed by the display of the virtualwindow device, i.e. the data base from which the section may be selectedis larger than the section itself. In this way, the display may becontrolled to switch viewing angles extremely quickly, simply byselecting different sections of the live video of the real scene. Forexample, the first camera unit may be a 3D-camera configured to capturea live 3D video of the real scene. In this way, the artificial windowdisplayed by the display may be rendered even more realistic.

According some embodiments of the invention, the distance sensor unitcomprises a plurality of sensor elements configured to produce each anindividual first distance signal indicating an individual distance tothe object. In particular, the sensor elements of the plurality ofsensor elements may be configured to produce the individual firstdistance signal such that it indicates an individual distance betweenthe object and a predetermined point for each sensor element, inparticular a predetermined point at each sensor element, for example anantenna of the sensor element.

The computing unit may be configured to calculate a position signalindicating a position of the object with respect to a reference pointbased on the individual first distance signals, for example, bytriangulation. The computing unit may be configured to select thesection of the live video of the real scene captured by the first cameraunit based on the position signal. In this way, a simple solution isprovided to determine, for example, with respect to which of a pluralityof users, the users being an example for the object, the virtual windowdevice should be controlled.

In the case that the first camera unit is a 3D camera, the section ofthe live 3D video of the real scene may be selected at differentdepth-of-view based on the calculated position signal.

According embodiments, the artificial window device comprises a secondcamera unit configured and arranged to capture a picture or a video of aspace in front of the display. The space in front of the display may beunderstood as a space which is reached by the light rays generated bythe display. In other words, a space faced by the display. The secondcamera unit may be configured and arranged to capture a solid angle ofthe space in front of the display, in particular directly in front ofthe display. In the event that the artificial window device is providedin an airplane, the second camera unit may be configured and arranged tocapture a picture at least, or solely, of a space in which a user ispresent when said user is seated on a seat of the airplane.

The artificial window device may comprise a face recognition unitconfigured to recognize a face in the captured picture or video of thespace in front of the display, to determine a spatial orientation of therecognized face and to produce a face orientation signal indicating atleast the determined orientation of the recognized face. The computingunit may be configured to select the section of the live video of thereal scene also based on the face orientation signal. In this way, thevirtual window displayed by the display may be changed according to atrue, or an approximated, viewing axis of the person whose face isrecognized, relative to a display.

A viewing axis may be defined as a viewing axis along which a person islooking at something, e.g. at the virtual window displayed by thedisplay.

According some embodiments, the first camera unit is arranged andconfigured to capture the live video of the real scene from a spacebehind the display. The space behind the display is to be understood asa space on the other side of the display as the space in front of thedisplay. In particular, when the display is arranged at an interior wallof a room or a vessel, the space before the display is the space withinsaid room or vessel, while the space behind the display is the space onthe outside of said room or said vessel, in particular on the other sideof the wall as the display. The space behind the display mayadvantageously include an area that would be visible to a person throughthe bezel in the hypothetical case that the bezel surrounded an openinginstead of the display.

The computing unit may be configured to select the section of the livevideo of the real scene captured by the first camera unit such that,based on the face orientation signal, the selected section is identicalto a hypothetical section that a person with a face with the determinedspatial orientation of the recognized face would see through the bezel,or the frame, if the bezel, or the frame, surrounded an opening or awindow instead of the display. In this way, an even more realisticvirtual window may be displayed to the user.

According to another embodiment, the face recognition unit is configuredto recognize a plurality of faces present in the picture or videocaptured by the second camera unit. A distance sensor unit may beconfigured to produce at least one first distance signal indicating arespective first distance to each of the recognized faces. Inparticular, the distance sensor unit may be configured to produce atleast one first distance signal indicating a respective first distancebetween the respective recognized face and at least one predeterminedpoint, in particular a predetermined point located at the distancesensor unit or, in the case that the distance sensor unit comprisesmultiple sensor elements, to each sensor element, in particular anantenna of each sensor element.

A computing unit may be configured to determine, based at least on theat least one first distance signal for each recognized face, a positionsignal indicating position of each recognized face with respect to acommon reference point. The position signal may simple compriseinformation about the distance of the face from the reference point, ormay comprise detailed information about the relative position of theface with respect to the reference point.

The computing unit may further be configured to select the section ofthe real scene captured by the first camera unit also based on theposition signals. In this way, for example, a prioritized list ofdifferent persons (to which the recognized faces belong) can be created.The virtual window displayed by the display can then be adapted tocomply with the viewing axis of the person at the top of the prioritizedlist.

According to another embodiment, the face recognition unit is configuredto produce a plurality of face orientation signals indicating a spatialorientation of the respective recognized face of each of the recognizedfaces. The computing unit may further be configured to select thesection of the live video of the real scene captured by the first cameraunit also based on the plurality of face orientation signals. In thisway, a prioritized list of the recognized faces may be created takinginto account that it may be more important to adapt the virtual windowto the true, or approximated, viewing axis of a person that is focusingon the display instead of adapting the virtual window to a person thatbarely grazes the display with its attention.

According some embodiments, the computing unit is further configured todetermine a preferred face among the plurality of recognized faces basedon at least the plurality of face orientation signals and/or on theplurality of position signals. The first camera unit may be arranged andconfigured to capture the live video of the real scene from the areabehind the display as described above. In this way, a prioritized listof the recognized faces may be created, taking into account that it maybe more important to adapt the virtual window displayed by the displayto the viewing axis of a person that is focusing on the display but islocated at a larger distance from the display instead of adapting thevirtual window to a person that barely grazes the display with itsattention but is located close to the display.

The computing unit may be configured to select the section of the livevideo of the real scene captured by the first camera unit such that,based on the face orientation signals belonging to the preferred face,the selected section is identical to a hypothetical section of the realscene that a person with a face having the spatial orientation of thepreferred face would see through the bezel, or the frame, if the bezel,or the frame, surrounded an opening or a window instead of the display.In this way, a weighting of the importance to display the virtual windowadapted to the various persons (to whom the recognized faces belong) maybe computed, taking into account both the proximity of the person to thedisplay (closer persons being generally preferred over more distantpersons) as well as the spatial orientation of the face of the person(orientations aimed closer at the display being generally preferred overorientations barely perceiving, or missing, the display in their fieldof view).

According to another embodiment, the computing unit is furtherconfigured to calculate, based on the plurality of face orientationsignals, a viewing angle between a viewing axis of each recognized faceand a predetermined main viewing axis of the display for each recognizedface. The predetermined main viewing axis of the display may be an axisperpendicular to the surface of the display and/or arranged at thegeometrical center of the display. The computing unit may be furtherconfigured to determine the preferred face based on the position signalsand on the calculated viewing angles.

According to some embodiments, the computing unit may be furtherconfigured to determine a pre-selection of recognized faces containingall recognized faces with a respective position signifying a seconddistance of the recognized faces from the reference point below apredetermined threshold distance value, based on the position signals.For example, the pre-selection may comprise all recognized faces at asecond distance of less than 5 meters from the reference point, whichmay be located at the display.

The computing unit may further be configured to determine the preferredface as the face within the pre-selection of recognized faces with thesmallest calculated viewing angle. In other words, in the above example,from the persons within 5 m of the display, the one person (or one face)is selected as preferred, which watches the display of the artificialwindow device most intently, or whose viewing axis coincides mostclosely with the main viewing axis of the display.

According to another embodiment, the computing unit is furtherconfigured to determine a pre-selection of recognized faces containingall recognized faces with a respective calculated viewing angle below apredetermined threshold angle value. For example, the pre-selection maycomprise all recognized faces that have a viewing axis arranged atthirty degrees or less with respect to the main viewing axis of thedisplay.

The computing unit may further be configured to determine the preferredface as that face within the pre-selection of faces which is located,based on the position signal, at the smallest second distance from thereference point.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be explained in greater detail with reference toillustrating embodiments depicted in the drawings as appended.

The accompanying drawings are included to provide a furtherunderstanding of the present invention and are incorporated in, andconstitute a part of, the specification. The drawings illustrate theembodiments of the present invention and together with the descriptionserve to explain the principles of the invention. Other embodiments ofthe present invention and many of the intended advantages of the presentinvention will be readily appreciated as they become better understoodby reference to the following detailed description. Elements of thedrawings are not necessarily to scale relative to each other. Likereference numerals designated corresponding similar parts.

FIG. 1 shows a schematic block diagram of a virtual window deviceaccording to an embodiment of the first aspect of the present invention;

FIG. 2 shows a schematic diagram illustrating the function of thevirtual window device of FIG. 1;

FIG. 3 shows a schematic flow diagram illustrating a method foroperating a virtual window device according to an embodiment of thesecond aspect of the present invention;

FIG. 4 shows a schematic block diagram of a virtual window deviceaccording to another embodiment of the first aspect of the presentinvention;

FIG. 5 shows a schematic diagram illustrating the function of thevirtual window device of FIG. 4;

FIG. 6 shows a schematic flow diagram illustrating a method foroperating a virtual window device according to another embodiment of thesecond aspect of the present invention;

FIG. 7 shows a schematic block diagram of a virtual window deviceaccording to yet another embodiment of the first aspect of the presentinvention;

FIG. 8 shows a schematic diagram illustrating the function of thevirtual window device of FIG. 7; and

FIG. 9 shows a schematic flow diagram illustrating a method foroperating a virtual window device according to yet another embodiment ofthe second aspect of the present invention.

In the figures, like reference numerals denote like or functionally likecomponents, unless indicated otherwise. Any directional terminology like“top”, “bottom”, “left”, “right”, “above”, “below”, “horizontal”,“vertical”, “back”, “front”, and similar terms are merely used forexplanatory purposes and are not intended to delimit the embodiments tothe specific arrangements as shown in the drawings.

DETAILED DESCRIPTION

Although specific embodiments have been illustrated and describedherein, it will be appreciated by those of ordinary skill in the artthat a variety of alternate and/or equivalent implementations may besubstituted for the specific embodiments shown and described withoutdeparting from the scope of the present invention. Generally, thisapplication is intended to cover any adaptations or variations of thespecific embodiments discussed herein.

FIG. 1 shows a schematic block diagram of a virtual window device 10according to an embodiment of the first aspect of the present invention.In the following, the virtual window device 10 will be explained usingan example wherein the virtual window device 10 is arranged in anairplane, in particular adjacent to an actual window or located betweentwo actual windows of the airplane. It is to be understood that thevirtual window device 10 may be adapted to, and used in, a large numberof other rooms or vehicles.

In FIG. 1, the virtual window device 10 is displayed with a bezel 15with a display 14 arranged within the bezel 15. The bezel 15 and thedisplay 14 may be arranged and configured such as to mimic theappearance of an actual window, for example an airplane window. Thevirtual window device 10 may, apart from the bezel 15 with the display14, comprise at least one actual window (not displayed), wherein thebezel 15 and the display 14 are configured and arranged such as to mimicthe appearance of said actual window. In particular, the display 14 maybe shaped in the same way as a transparent portion, for example a glassplate, arranged in the actual window.

Although here and in the following, reference is made mainly to a bezel15, it should be understood that all virtual window devices describedherein may be provided with a frame instead of the bezel 15, wherein thedisplay 14 is arranged within the frame. Especially when the virtualwindow device 10 is arranged in a room of a building, the frame with thedisplay 14 may be formed as to mimic the appearance of a window framesurrounding an actual window.

The virtual window device 10 may also be provided with a shutterconfigured to enable a user to partially or completely cover the bezel15 and/or the display 14 with the shutter. In particular, the shuttermay be configured and arranged with respect to the bezel 15 and thedisplay 14 in the same way as a shutter of the actual window locatednext to the bezel 15 and the display 14.

The virtual window device 10 further comprises a first camera unit 18configured to capture a live video of a real scene 30. In FIG. 1, as anexample of a real scene, a cloudy sky is shown.

The first camera unit 18 is, in particular, configured to capture thelive video of the real scene for a space behind the bezel 15 with thedisplay 14. The virtual window device 10 may comprise a wall, or a wallsection, in which the bezel 15 and the display 14 are arranged. Thefirst camera unit 18 may be positioned and configured such that the livevideo of the real scene 30 is captured from a space on the side of saidwall which is opposite to the side of said wall on which the bezel 15and the display 14 are arranged. In the event that the virtual windowdevice 10 is arranged on, or as part of, an aircraft, the first cameraunit 18 may be arranged to capture live video from the outside of theairplane. In this way, a virtual window can be displayed to personsinside the airplane (e.g. passengers in a passenger cabin and/or a pilotin a cockpit of the airplane) that has the appearance of an actualwindow showing the real scene 30 in the same way as the actual windowssurrounding, or adjacent to, the bezel 15 with the display 14, allowpersons to look outside.

The first camera unit 18 may be a single camera device or may compriseseveral individual camera devices. The fields of view of the individualcamera devices may overlap or at least be adjacent to each other. Inthis way, the entirety of the camera devices of the first camera unit 18may be configured to capture the live video of the real scene 30. Knowntechniques for stitching together different views from a plurality ofindividual camera devices may be applied to obtain a singlenon-redundant live video of the real scene 30, e.g. by a CPU of thefirst camera unit 18.

The first camera unit 18 may be a 3D camera. Also in this case, thefirst camera unit may comprise, or consist of, a plurality of individualcamera devices which may be configured to capture different aspects ofthe real scene 30, e.g. with different depths of field. Known techniquesfor creating 3D video data may be applied to create a live 3D videofield of the real scene 30.

The virtual window device 10 further comprises a distance sensor unit 16configured to produce at least one first distance signal 91 indicating afirst distance 51 to an object 61. The distance sensor unit may compriseone or more sensor elements which may be configured as ultrasound sensorelements, light sensor elements or other known sensor elements fordetermining distances to objects. Each sensor element of the distancesensor unit 16 may be configured to produce one individual firstdistance signal 91 indicating the first distance 51 between therespective sensor element and the object 61.

The object 61 to which the first distance 51 is determined and indicatedcan be an object with a predetermined pattern, in particular a person ora part of a person, especially a face of a person. In the following, theinvention will be described with reference to a face 61 of a person asthe object. It should be understood that the methods and devicesdescribed herein may also be used with other objects or types ofobjects, for example, camera lenses.

The distance sensor unit 16, or at least one sensor element of thedistance sensor unit 16, or at least one antenna of the distance sensorunit 16, can be arranged adjacent to the bezel 15 or the display 14, forexample arranged within the display 14, in particular at a geometricalcenter of the display 14. In this way, an especially close relationbetween the first distance 51 between the face 61 and the distancesensor unit 16 or one of its sensor elements on the one hand, and thedistance of the face 61 from the artificial window created by thedisplay 14 on the other hand is established.

The virtual window device 10 further comprises a computing unit 20configured to select, based at least on the at least one first distancesignal 91, a section 32 of the live video of the real scene 30 capturedby the first camera unit 18. For this, the first camera unit 18 isconfigured to create, and transmit to the computing unit 20, a videosignal 98 corresponding to the live video of the real scene 30.

The computing unit 20 is configured to control the display 14 to displaythe selected section 32 of the real scene 30, that is, the display 14 iscontrolled to display only the selected section 32. In order to controlthe display 14, the computing unit 20 creates a control signal 99 andtransmits the control signal 99 to the display 14.

The computing unit 20 is configured to select the section 32 based onthe measured first distance 51. In particular, the computing unit 20 maybe configured to select a larger section 32 from the captured live videoof the real scene 30 if the first distance 51 to the face 61 is smallerand to select a smaller part of the captured live video of the realscene 30 as the section 32 when the distance 51 to the face 61 islarger. In this way, the display 14 is controlled to create for a personwith the face 61 the impression of gaining a wider solid angle of view34 through the artificial window when coming closer to the artificialwindow, just as a solid angle of view 34 through an actual window of theperson would widen when said person came closer to that actual window.

In the case that the virtual window device 10 is equipped with ashutter, the computing unit 20 may be configured to shut the display 14off in the event that the shutter is closed completely or at least to apredetermined degree.

FIG. 2 shows a schematic diagram illustrating the function of thevirtual window device 10 according to FIG. 1.

As an example, in FIG. 2, the bezel 15 with the display 14 is shown asarranged within, or attached to, a wall 2 of an airplane 1 (not shown inits entirety), facing inward. Also the first camera unit 18 is arrangedwithin, or attached to, the external wall 2 of the airplane 1, facingoutward. FIG. 2 illustrates that, although the first camera unit 18captures a scene 30 within a comparatively large solid angle, thedisplay 14 is controlled so as to display only the selected section 32which the person with the face 61 would perceive if the bezel 15surrounded the window of the airplane 1. The bezel 15 may be providedwith glass plates, in particular with glass plates equal to glass platesof an adjacent actual window in front of the display 14, in order tocreate a more convincing simulation of an actual window and to protectthe display 14 from harm.

FIG. 3 shows a schematic flow diagram illustrating a method foroperating a virtual window device according to another embodiment of theinvention. The method illustrated in FIG. 3 may be performed for exampleby, or for, or with the artificial window device 10 of FIG. 1 and FIG. 2and may be adapted to all variations and modifications described with orfor the artificial window device 10 in the foregoing.

In a method step S01, a distance 51 between an object, in particular aperson, especially a face 61 of a person, and a reference point isdetermined, e.g. as described above with respect to the distance sensorunit 16. In a method step S02, a live video of a real scene 30 iscaptured, e.g. as described above with respect to the first camera unit18.

In a method step S03, based at least on the determined distance 51, asection 32 of the captured live video of the real scene 30 is selected,e.g. as described above with respect to the computing unit 20. In amethod step S04, a display 14 of a virtual window device 10 iscontrolled to display the selected section 32, e.g. as described abovewith respect to the computing unit 20.

FIG. 4 shows a schematic block diagram of a virtual window device 110according to another embodiment of the first aspect of the presentinvention. The virtual window device 110 is a variant of the virtualwindow device 10 and is adaptable to any modifications or variationsdescribed in the foregoing with respect to the virtual window device 10.The virtual window device 110 will also be described with respect to thefollowing FIG. 5.

The virtual window device 110 is different from the virtual windowdevice 10 in particular in a distance sensor unit 116 of the virtualwindow device 110 replacing the distance sensor unit 16 of the virtualwindow device 10 and in a computing unit 120 of the virtual windowdevice 110 replacing the computing unit 20 of the virtual window device10.

The distance sensor unit 116 comprises a plurality of sensor elements117-1, 117-2 configured to produce each an individual first distancesignal 191-1, 191-2 indicating an individual distance 154-1, 154-2 tothe face 61, respectively. Although in FIGS. 3 and 4 two sensor elements117-1, 117-2 are shown, for example three or more sensor elements areprovided to enable, or improve, a triangulation of the face 61.

The computing unit 120 comprises a position calculation module 122. Theposition calculation module 122 is configured to calculate a positionsignal 192 indicating a position of the face 61 with respect to areference point 172, based on the individual first distance signals191-1, 191-2, for example by triangulation. The computing unit 120 maybe realized with a processor and a memory and be configured byconfiguring software that the computing unit 120 is set to execute.

The reference point 172 is arranged, in the embodiment of FIG. 4, at ageometrical center of the display 14, e.g. its center of mass. The firstand the second sensor elements 117-1, 117-2 may be advantageouslyprovided at least at two different sides of the bezel 15, e.g. at twoopposite sides, or on all sides of the bezel 15, in particularsymmetrically with respect to the reference point 172. In this way, theposition of the face 61 with respect to the reference point 172 may bedetermined with better precision.

The computing unit 120 comprises a selection module 124. The selectionmodule 124 is configured to select the section 32 of the live video ofthe real scene 30 captured by the first camera unit 18 based, at leastin part, on the calculated position signal 192, i.e. based on theposition of the face 61.

The virtual window device 110 thus enables an even more precise anddynamically adapted selection of the section 32 such that the virtualwindow displayed by the display 14 changes in the same way, when aperson comes closer to the display 14, as the visible section of anactual window would change for a person when that person came closer tothe actual window, and similarly for a person moving away from thedisplay 14.

Alternatively or additionally, the computing unit 120 may be configuredto select the section 32 such that the virtual window displayed by thedisplay 14 is identical, or very similar to, what an actual windowlocated at the position of the display 14 would show to a person with aface 61 located at the position indicated by the position signal 192.The section 32 can be dynamically updated according to changes in theposition signal 192 indicating changes of the position of the face 61.In other words, the solid angle with which the section 32 is selectedmay be chosen such as to correspond to the solid angle of view 34 of theface 61 located at the position indicated by the position signal 192.That solid angle of view 34 may be computed by taking the positionindicated by the position signal 192 as a vertex of a solid angledelimited by a contour of the display 14.

Alternatively, or in addition, the selection module 124 may beconfigured to select the section 32 based on the plurality of individualfirst distance signals 191-1, 191-2. In this case, the plurality ofindividual first distance signals 191-1, 191-2 taken together may betermed the position signal 192.

The position calculation module 122 and/or the selection module 124 maye.g. be configured with separate processors and/or memories, or they maybe realized as software modules of software executed by the computingunit 120.

FIG. 5 shows a schematic diagram illustrating the function of thevirtual window device 110 according to FIG. 4.

In FIG. 5, a first sensor element 117-1 of the distance sensor unit 116and a second sensor element 117-2 of the distance sensor unit 116 areshown as arranged on two opposite sides of the bezel 15, in particular,the first sensor element 117-1 above the bezel 15 and the second sensorelement 117-2 below the bezel 15. When the face 61 (symbolized by asingle eye) is located somewhat upwards of the bezel 15, for examplewhen a person is standing in front of the bezel 15, the first distancesignal 191-1 produced by the first sensor element 117-1 indicates acomparatively shorter individual first distance 154-1, whereas the firstdistance signal 191-2 produced by the second sensor element 117-2indicates a comparatively individual first distance 154-2.

The position calculation module 122 may be configured to determine inthis situation that the face 61 is located at a position upwards of thereference point 172 so that the section 32 displayed by the display 14may be chosen corresponding to a viewing angle originating from abovethe display 14, as described above.

In addition or alternatively, a viewing axis 36 of the person with theface 61 may be calculated by the computing unit 120, in particular bythe position calculation module 122. The viewing axis 36 may becalculated as a line passing through the position of the face 61indicated by the position signal 192 and through the reference point172, especially through the geometrical center of the display 14. Fromthe viewing axis 36 and a known main viewing axis of the display 14,which can extend perpendicularly to the display 14 from the referencepoint 172, a viewing angle of the face 61 may be calculated, cf. alsoFIG. 8 for illustration.

The selection module 124 may be configured to select the section 32 alsobased on the viewing angle. In the event that multiple faces 61 aredetermined before the display 14, the section 32 may be based on theviewing angle of the face 61 with the smallest viewing angle.

FIG. 6 shows a schematic flow diagram illustrating a method foroperating a virtual window device according to another embodiment of thesecond aspect of the present invention. The method illustrated in FIG. 6may be performed for example by the artificial window device 110 of FIG.3 and FIG. 4 and may be adapted to all variations and modificationsdescribed with for the artificial window device 110 in the foregoing.

The method described with reference to FIG. 6 is a variant of the methoddescribed with reference to FIG. 3 and differs from that method by amodified step S103 instead of the step S03 and by comprising thefollowing additional method steps:

In a method step S05, a plurality of first distances 154-1, 154-2 to theobject 61 is determined. In particular, the plurality of first distancesis a plurality of distances between the object 61 and predeterminedpoints, wherein the predetermined points 117-1, 117-2 can be locationsof sensor elements 117-1, 117-2. The determining S05 of the plurality offirst distances 154-1, 154-2 may be conducted as described above withreference to the distance sensor unit 116. The object 61 may inparticular be a person's face.

In a method step S06, a position of the object 61 is determined, forexample calculated, based on the plurality of measured first distances154-1, 154-2, for example as described above with respect to thecomputing unit 120 and the position calculation module 122.

In the method step S103, the section 32 of the captured live video ofthe real scene 30 is selected based on the determined position of theobject 61, for example, as described above with respect to the computingunit 120 and the selection module 124.

FIG. 7 shows a schematic block diagram of a virtual window device 210according to yet another embodiment of the first aspect of the presentinvention. The virtual window device 210 will also be described withrespect to the following FIG. 8. The virtual window device 210 is avariant of the virtual window device 110 and is adaptable to anymodifications or variations described in the foregoing with respect tothe virtual window device 110.

The virtual window device 210 is different from the virtual windowdevice 110 in particular in a computing unit 220 of the virtual windowdevice 210 replacing the computing unit 120 of the virtual window device110, and in that the virtual window device 210 comprises a second cameraunit 219.

The second camera unit 219 is configured and arranged to capture apicture or a video of a space in front of the display 14. Continuing theexample from the previous figures, wherein the display 14 was formed orarranged in, or attached to, a wall 2 of an airplane 1, the secondcamera unit 219 may be configured and arranged to capture a picture or avideo from a space inside the airplane 1, in front of the display 14.The picture or video captured by the second camera unit 219 istransmitted to a face recognition module 226 of the computing unit 220in a data signal 297.

The face recognition module 226 is configured to recognize at least oneface 261, 262 in the captured picture or video of the space in front ofthe display 14 based on the data signal 297, as is visualized in FIG. 8.The face recognition module 226 may transmit information indicating therecognized faces 261, 262, for example to a position calculation module222 of the computing unit 220 in a data signal 296. The positioncalculation module 222 may be configured to process the individual firstdistance signals 191-1, 191-2 in the same way as the positioncalculation module 122 of the computing unit 120 of the virtual windowdevice 110. The position calculation module 222 may further beconfigured to determine, based on the data signal 296 indicating therecognized faces 261, 262, the positions of all of the recognized faces261, 262 based on the plurality of individual first distance signals191-1, 191-2.

The face recognition module 226 may further be configured to determine aspatial orientation of the at least one recognized face 261, 262 and toproduce a face orientation signal 293 indicating at least the determinedspatial orientation of the at least one recognized face 261, 262. Theface recognition module 226 may, for example, comprise a processor and amemory, or it may be realized as part of the computing unit 220, e.g. asa software module of a software executed by the computing unit 220.

The computing unit 220 may be configured as described with respect tothe computing unit 120 of the virtual window device 110 and may furtherbe configured to select the section 32 of the live video of the realscene 30 captured by the first camera unit 18 also based on the faceorientation signal 293. In particular, the determined face orientationof the recognized faces 261, 262 may replace, or supplement, theinformation about the viewing angle of the person with the face 261.

Thus, the a selecting module 224 of the computing unit 220, which may inother respects be configured in the same way as the selecting module 124of the computing unit 120 of the virtual window device 110, may beconfigured to select the section 32 based on, for example:

-   -   the distance of the face 261, 262 from the reference point 172,        in particular from a geometrical center of the display 14, and        the face orientation signal 293 indicating the respective        orientation of the face 261, 262;    -   the plurality of first distance signals 191-1, 191-2 and the        face orientation signal 293 indicating the face orientation of        the face 261, 262;    -   the distance of the face 261, 262 from the reference point 172        and/or the plurality of first distance signals 191-1, 191-2        only, for example in the event that a determination of the face        orientation of the recognized faces 261, 262 has failed.

The selection module 224 may then, based on the above and/or as has beendescribed in detail with respect to the selection module 124 of thecomputing unit 120 of the virtual window device 110, select the section32 such that the display 14 displays a virtual window that is identical,or very similar, to a section of the real scene 30 that the person withthe face 261, 262 would see through an actual window located at theposition and with the shape of the display 14.

In the event that a plurality of faces 261, 262 is recognized by theface recognition module 226, the selection module 224 may further beconfigured to determine a preferred face 261 among the recognized faces,and to select the section 32 such that the display 14 displays a virtualwindow that is identical, or very similar, to a section of the realscene 30 that the person with the preferred face 261 would see throughan actual window located at the position and with the shape of thedisplay 14. The preferred face 261 may be selected based on e.g. itsdistance from the reference point 172 and/or on its viewing angle (basedon the individual first distance signals 191-1, 191-2 and/or on the faceorientation signal 293).

In order to determine the preferred face 261, the selection module 224may first create a pre-selection of faces 261, 262.

For example, the selection module 224 may be configured to firstcalculate, based on the face orientation signals 293 for the recognizedfaces 261, 262 (and/or on the individual first distance signals 191-1,191-2), a viewing angle 39 between a viewing axis 36 of each recognizedface 261, 262 and a predetermined main viewing axis 38 of the display 14for each recognized face 261, 262, as illustrated in FIG. 8. Theselection module 224 may further be configured to calculate a seconddistance of each recognized face 261, 262 from the reference point 172,e.g. based on the position signal 292 for each recognized face 261, 262and/or based on the individual first distance signals 191-1, 191-2.

As a first example, the selection module 224 may be further configuredto determine the pre-selection of recognized faces 261, 262 ascontaining all recognized faces 261, 262 with a respective calculatedviewing angle 39 below a predetermined threshold angle value, forexample, sixty degrees, forty-five degrees, or thirty degrees. Theselection module 224 may be further configured to then determine thepreferred face 261 as the face within the pre-selection of recognizedfaces 261, 262 which is located at the smallest second distance from thereference point 172.

As a second example, the selection module 224 may be further configuredto determine the pre-selection of recognized faces 261, 262 ascontaining all recognized faces 261, 262 with a second distance from thereference point 172 below a predetermined, or adjustable, thresholddistance value, based on their respective position signals 292. Forexample, the threshold distance value may be five meters, or one meter.The selection module 224 may be further configured to then determine thepreferred face 261 as the face within the pre-selection of recognizedfaces 261, 262 with the smallest calculated viewing angle 39. Thissecond example is schematically illustrated in FIG. 8.

FIG. 8 shows a schematic diagram illustrating the function of thevirtual window device 210 of FIG. 7. If it is assumed that both a firstface 261 and a second face 262 have been recognized and entered into thepre-selection of recognized faces, the selection module 224 may then,based on the calculated viewing angles 39 of the faces 261, 262, decidewhich one is the preferred face, and control the display 14 to display asection 32 based on the viewing angle 39, or viewing axis 36, of theperson with the preferred face.

In FIG. 8 it is assumed that the person with the first face 261 islooking relatively directly at the display 14, such that it has aviewing angle 39 with respect to the main viewing axis 38 of the display14 of about 20 degrees. The person with the second face 262 only barelyperceives the display 14 at all. Accordingly, the first face 261 isselected as the preferred face and the display 14 is controlled todisplay the section 32 based on parameters (distances, viewing angle) ofthe first face 261.

FIG. 9 shows a schematic flow diagram illustrating a method foroperating a virtual window device according to yet another embodiment ofthe second aspect of the present invention. The method illustrated inFIG. 9 may be performed for example by the artificial window device 210of FIG. 7 and FIG. 8 and may be adapted to all variations andmodifications described with for the artificial window device 210 in theforegoing.

The method described with reference to FIG. 9 is a variant of the methoddescribed with reference to FIG. 6 and differs from that method by amodified step S203 instead of the step S103 and by comprising thefollowing additional method steps:

In a method step S07, a picture or a video of a space in front of thedisplay 14 is captured, e.g. as described above with respect to thesecond camera unit 219 of the virtual window device 210.

In a method step S08, at least one face 261, 262 within the capturedpicture or video of the space in front of the display 14 is recognized.In a method step S09, a spatial orientation of the at least onerecognized face 261, 262 is determined. Step S08 and/or step S09 may beperformed e.g. as described above with respect to the face recognitionmodule 226 of the virtual window device 210.

In a method step S203, replacing the method step S103 of the method ofFIG. 6, a section 32 of the captured live video of the real scene 30 isselected based at least on the spatial orientation of the at least onerecognized face 261, 262, in addition to based on the determinedposition of the at least one recognized face 261, 262.

A plurality of faces 261, 262 may be recognized in the captured pictureor video of the space in front of the display 14. For each recognizedface 261, 262, a respective distance between the recognized face 261,262 and the reference point 172 may be calculated, e.g. based on therespective position signal 292 and/or directly on the individual firstdistance signals 191-1, 191-2.

The method may comprise some, or all of, the following additional methodsteps:

In a method step S10, a viewing angle between a viewing axis 36 of eachof the recognized faces 261, 262 and a predetermined main viewing axis38 of the display 14 for each of the recognized faces 261, 262 iscalculated. In a method step S11, a preferred face 261 is determinedbased on the calculated distances between the recognized faces 261, 262and the reference point 172, as well as based on the calculated viewingangles 39.

In the method step S203, the section 32 of the captured live video ofthe real scene 30 may be selected as identical to a hypothetical sectionthat a person with a face having the spatial orientation of thepreferred face 261 would see through a bezel 15, or a frame, surroundingthe display 14 if the bezel 15, or the frame, surrounded an opening or awindow instead of the display 14.

The method steps S10, S11 and S203 may be performed e.g. as describedwith respect to the selection module 224 of the virtual window device210 and/or with respect to the selection module 124 of the virtualwindow device 110.

While at least one exemplary embodiment of the present invention(s) isdisclosed herein, it should be understood that modifications,substitutions and alternatives may be apparent to one of ordinary skillin the art and can be made without departing from the scope of thisdisclosure. This disclosure is intended to cover any adaptations orvariations of the exemplary embodiment(s). In addition, in thisdisclosure, the terms “comprise” or “comprising” do not exclude otherelements or steps, the terms “a” or “one” do not exclude a pluralnumber, and the term “or” means either or both. Furthermore,characteristics or steps which have been described may also be used incombination with other characteristics or steps and in any order unlessthe disclosure or context suggests otherwise. This disclosure herebyincorporates by reference the complete disclosure of any patent orapplication from which it claims benefit or priority.

1. A virtual window device comprising: a bezel, or a frame, with adisplay arranged within the bezel, or within the frame; a first distancesensor unit configured to produce at least one first distance signalindicating a first distance to an object and arranged adjacent to thebezel or the frame or arranged within the display; a first camera unitconfigured to capture a live video of a real scene; and a computing unitconfigured to select, based at least on the at least one first distancesignal, a section of the live video of the real scene captured by thefirst camera unit; wherein the computing unit is further configured tocontrol the display to display the selected section.
 2. The device ofclaim 1, wherein the distance sensor unit comprises a plurality ofsensor elements configured to produce each an individual first distancesignal indicating an individual distance to the object, respectively;wherein the computing unit is configured to calculate a position signalindicating a position of the object with respect to a reference point,based on the individual first distance signals; and wherein thecomputing unit is configured to select the section of the live video ofthe real scene captured by the first camera unit based on the calculatedposition signal.
 3. The device of claim 1, further comprising: a secondcamera unit configured and arranged to capture a picture or a video of aspace in front of the display; and a face recognition module configuredto recognize a face in the captured picture or video of the space infront of the display, to determine a spatial orientation of therecognized face and to produce a face orientation signal indicating atleast the determined spatial orientation of the recognized face; whereinthe computing unit is further configured to select the section of thelive video of the real scene captured by the first camera unit alsobased on the face orientation signal.
 4. The device of claim 1, whereinthe first camera unit is arranged and configured to capture the livevideo of the real scene from a space behind the display; and wherein thecomputing unit is configured to select the section of the live video ofthe real scene captured by the first camera unit such that, based on theface orientation signal, the selected section is identical to ahypothetical section that a person with a face with the spatialorientation of the recognized face would see through the bezel, or theframe, if the bezel, or the frame, surrounded an opening or a windowinstead of the display.
 5. The device of claim 3, wherein the facerecognition module is configured to recognize a plurality of faces inthe picture or video captured by the second camera unit; wherein thedistance sensor unit is configured to produce at least one firstdistance signal indicating a respective first distance to each of therecognized faces; wherein the computing unit is configured to determine,based at least on the at least one first distance signal for eachrecognized face, a position signal indicating a position of eachrecognized face with respect to a common reference point; and whereinthe computing unit is further configured to select the section of thereal scene captured by the first camera unit also based on the positionsignals.
 6. The device of claim 5, wherein the face recognition moduleis configured to produce a plurality of face orientation signalsindicating a spatial orientation of the respective recognized face;wherein the computing unit is further configured to select the sectionof the live video of the real scene captured by the first camera unitalso based on the plurality of face orientation signals.
 7. The deviceof claim 6, wherein the computing unit is further configured todetermine a preferred face among the plurality of recognized faces basedon at least one of the plurality of face orientation signals and on theplurality of position signals; wherein the first camera unit is arrangedand configured to capture the live video of the real scene from an areabehind the display; and wherein the computing unit is configured toselect the section of the live video of the real scene captured by thefirst camera unit such that, based on the face orientation signalbelonging to the preferred face, the selected section is identical to ahypothetical section that a person with a face having the spatialorientation of the preferred face would see through the bezel, or theframe, if the bezel, or the frame, surrounded an opening or a windowinstead of the display.
 8. The device of claim 7, wherein the computingunit is further configured to calculate, based on the plurality of faceorientation signals, a viewing angle between a viewing axis of eachrecognized face and a predetermined main viewing axis of the display foreach recognized face; and wherein the computing unit is furtherconfigured to determine the preferred face based on the position signalsand on the calculated viewing angles.
 9. The device of claim 8, whereinthe computing unit is further configured to determine a pre-selection ofrecognized faces containing all recognized faces with a second distanceof each recognized face from the reference point below a predeterminedthreshold distance value, based on the position signals; and wherein thecomputing unit is further configured to determine the preferred face asthe face within the pre-selection of recognized faces with the smallestcalculated viewing angle.
 10. The device of claim 8, wherein thecomputing unit is further configured to determine a pre-selection ofrecognized faces containing all recognized faces with a respectivecalculated viewing angle below a predetermined threshold angle value;and wherein the computing unit is further configured to determine thepreferred face as that face within the pre-selection of faces which islocated, based on its position signal, at the smallest second distancefrom the reference point.
 11. A vehicle with a virtual window device,the device comprising: a bezel, or a frame, with a display arrangedwithin the bezel, or within the frame; a distance sensor unit configuredto produce at least one first distance signal indicating a firstdistance to an object and arranged adjacent to the bezel or the frame orarranged within the display; a first camera unit configured to capture alive video of a real scene; and a computing unit configured to select,based at least on the at least one first distance signal, a section ofthe live video of the real scene captured by the first camera unit;wherein the computing unit is further configured to control the displayto display the selected section.
 12. A method for operating a virtualwindow device, comprising: determining a distance between an object anda reference point using a distance sensor and arranged adjacent to abezel or a frame or arranged within a display; capturing a live video ofa real scene; selecting, based at least on the determined distance, asection of the captured live video of the real scene; and controllingthe display of the virtual window device to display the selectedsection.
 13. The method of claim 12, further comprising: determining aplurality of first distances to the object; and determining a positionof the object based on the plurality of measured first distances;wherein the selecting of the section of the captured live video of thereal scene is based also on the determined position of the object. 14.The method of claim 12, further comprising: capturing a picture or avideo of a space in front of the display; recognizing at least one facewithin the captured picture or video of the space in front of thedisplay; and determining a spatial orientation of the at least onerecognized face; wherein the selecting of the section of the capturedlive video of the real scene is based also on the spatial orientation ofthe at least one recognized face.
 15. The method of claim 14, wherein aplurality of faces is recognized in the captured picture or video of thespace in front of the display; and wherein, for each recognized face, arespective distance between the recognized face and the reference pointis calculated; and wherein the method further comprises: calculating aviewing angle between a viewing axis of each of the recognized faces anda predetermined main viewing axis of the display for each of therecognized faces; determining a preferred face based on the calculateddistances between the recognized faces and the reference point as wellas based on the calculated viewing angles; and wherein the selecting ofthe section of the captured live video of the real scene is selected asidentical to a hypothetical section that a person with a face having thespatial orientation of the preferred face would see through a bezel, ora frame, surrounding the display if the bezel, or the frame, surroundedan opening or a window instead of the display.
 16. The device of claim1, wherein the distance sensor is arranged at a geometric center of thedisplay.
 17. The device of claim 1, further comprising a second distancesensor, wherein the first distance sensor and second distance sensor arearranged on two different sides of the bezel.